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A  compiler-based  specification  and  testing 
system  for  defining  data  types  has  been  developed* 
The  system,  DA1STS,  includes  formal  algebraic 
specifications  and  statement  and  expression  test 
coverage  monitors*  This  paper  descibes  our  initial 
attempt  to  evaluate  the  effectiveness  of  the  system 
in  helping  users  produce  software  containing  fewer 
errors.  In  an  exploratory  study,  subjects  without 
prior  experience  with  DA1STS  were  encouraged  by  the 
system  to  develop  effective  sets  of  test  cases  for 
their  implementations*  Furthermore,  an  analysis  of 
the  errors  remaining  in  the  implementat  ions  provided 
valuable  hints  about  additional  useful  testing 
metrics.. 
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1 •  Int  roduct ion 


Program  development  remains  an  error-prone  process* 
Specifications  are  often  ambiguous  or  incomplete*  and  validation 
of  a  program's  conformance  to  its  specification  by  testing  is 
often  performed  by  humans  who  agree  too  readily  with  the  output 
of  the  test  program  and  who  have  little  feel  for  how  thoroughly 
the  specification  or  program  has  been  tested.  These  problems  are 
magnified  during  the  later  stages  of  the  software  life  cycle. 
Since  specifications*  programs  and  test  data  are  usually  separate 
entities*  each  can  be  altered  without  regard  to  the  others. 
Furthermore,  test  data  collected  because  it  exposes  a  particular 
error  is  theoretically  useless  as  soon  as  the  error  it  exposes  is 
corrected. 

Program  testing  systems  have  been  developed  that  compare 
user-supplied  and  program-computed  input-output  pairs*  and 
measure  several  program  coverage  criteria  (e.g.*  statements, 
paths*  expression  values*  etc.)  CHamtet  78T.  These  input-output 
pairs  can  be  difficult  to  write  for  complex  functions  (e.g.*  the 
result  of  adding  a  single  identifier  to  a  hash-coded  symbol 
table)*  and  when  an  error  is  detected*  the  user-supplied  pair  is 
often  as  suspect  as  the  computed  one.  Testing  systems  whose 
criteria  for  test  data  selection  involve  only  program  structure 
are  too  weak  to  reveal  all  design  errors  and  many  types  of 
construction  errors  CGoodenough  and  Gerhart  751. 

We  have  combined  recent  work  in  data  abstraction 
specification  and  modularization  with  a  program  testing  system  in 
an  attempt  to  ease  program  development.  OAISTS  (Data  Abstraction 
Implementation*  Specification*  and  Testing  System) 
[Gannon*  et  al.  803  combines  a  data  abstraction  language 
containing  SIMULA-like  classes  [Dahl*  et  al.  683  and  algebraic 
specifications  similar  to  those  of  CGuttag  773  with  a  library  of 
test  monitoring  routines.  with  user-supplied  test  sets,  the 
axioms  of  the  specification  are  used  as  driver  orograms  for  the 
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implementation.  Structural  testing  criteria  are  applied  to  both 
axioms  and  code  to  evaluate  the  test  data*  We  feel  OAISTS  has 
several  advantages  over  conventional  program  development  systems: 


1)  The  specification*  program*  and  test  <?ata  are 
packaged  as  a  single  entity*  encouraging  their  mutual 
maintenance* 

2)  The  specification  language  is  apolicative  and  the 
implementation  language  is  imperative*  We  hope  that  this 
orthogonality  will  reduce  the  liklihood  of  the  same  error 
appearing  in  both  the  specification  and  the  implementation. 

3)  The  test  data  coverage  of  the  specification  and  the 
program  are  measured. 

4)  There  is  no  need  to  describe  the  concrete 
representation  produced  by  an  operation;  the  user 
(specifies  and)  writes  an  equality  routine  to  judge  the 
results  of  tests  for  abstract  objects.  This  simplifies  the 
testing  process  by  removing  the  requirement  for  'hand 
simulation'  of  complicated  operations* 

5)  Having  a  tool  that  incorporates  specifications  into 
the  development  process  should  provide  the  motivation  and 
experience  necessary  for  proarammers  to  use  formal 
specifications  effectively. 


The  construction  of  every  software  tool  should  include  an 
evaluation  of  its  effectiveness*  The  evaluation  can  be  used  to 
convince  users  of  a  system's  worth  and  can  also  provide  useful 
information  to  the  designers  about  its  shortcomings*  This  paper 
describes  an  exploratory  study  that  compared  program  development 
with  DA  I  STS  against  more  conventional  programming  techniques*  We 
felt  that  structure  imoosed  on  the  programming  process  by  daISTS 
would  aid  programmers  in  constructing  programs  containing  fewer 
delivered  errors*  but  were  concerned  about  program  development 
costs  and  the  ability  of  users  to  adaot  to  DAISTS* 
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2.  OAISTS 


A  program  submitted  to  DAISTS  contains  an  implementation  of 
an  abstract  data  type  written  in  the  high  level  language  SIMPL-D 
CGannon  and  Rosenberg  791?  a  collection  of  algebraic  axioms 
describing  the  type?  and  a  collection  of  test  cases* 

2*1*  Implementation  Language 

SIMPL-D  £la§s  declarations  define  new  types  which  may 
subsequently  be  used  in  object  declarations*  The  inferior  o*  * 
cl^ass  is  a  series  of  variable  declarations  (the  representation) 
followed  by  a  series  of  procedure  declarations  (the  body). 
Including  the  names  of  procedures  in  the  cl,ass  heading  makes  the 
procedures  visible  outside  the  £ia§s*  The  appearance  of  the 
reserved  word  ajjign  in  the  operation  list  enables  the  assignment 
operation  ( :  = )  to  be  applied  to  objects  of  this  type* 

When  a  unit  of  scope  containing  the  declaration  of  a  £l£££ 
object  is  executed?  a  new  copy  of  the  c^ajs  object  is  allocated 
and  initialized*  Users  generally  view  c±aj§  objects  as 
indivisible  entities;  the  components  of  £la££  objects  cannot  be 
accessed  outside  of  its  c 55  declaration*  Inside  the  £l22i? 
these  oojects  may  be  viewed  as  structures  containing  more 
primitive  objects*  Statements  in  the  body  of  a  can  access 
the  yQ2SU£  components  of  a  c^ajj  object  using  a  period  notation 
similar  to  that  of  PL/1  or  Pascal. 

A  fragment  of  an  implementation  for  bounded  lists  of 
integers  is  shown  below.  Each  list  object  is  represented  by  a 
boolean  variable  (Empty)  indicating  whether  or  not  the  list  is 
empty?  an  array  of  integers  (Values)  holding  the  values  currently 
in  the  list?  and  integer  indices  (Head  and  Tail)  into  the  array 
identifying  the  first  and  last  elements* 


£1222  List  a  NewList*  AddFirst*  DeleteFirst*  DeleteLast* 
ListLength*  ...*  222190 

f*  The  representat ion  */ 

dStlQS  ListSize  =  'll" 

UQlSyf  fefifilSiQ  Empty 
ynigye  Tnf  Ready  Tail 

UQjgyfi  I5l  iEZai  Values(ListSize) 

/*  The  operations  */ 

List  fyne  NewList  /*  returns  list  with  no  elements  •/ 
List  Result 
Result. Empty  :*  t£y£ 

££iy£Q  CResul t) 

List  Tyne  De let eF i rst (Li st  x  )  /*  returns  Tail(X)  */ 
List"Result 
Result  :*  x 

if  Result. Empty  ar  Result. Head  *  Result. Tail 
£hen  /*  list  contains  zero  or  one  element  */ 
retyrn  (NewList) 

else  7*  increment  Head  modulo  ListSize  */ 

“""If  Result. Head  <>  ListSize  -  1 
then 

Result. Head  : =  Result. Head  ♦  1 
e^2e  circularize  to  zero  origin  */ 
Result. Head  :=  0 

eng 

££IM£Q (Result ) 

ssg 


sodsiaii 


2 .2.  Specif icat ion  Language 


The  specification  language  for  DAXSTS  is  similar  to  that 
described  in  CSuttag  et  al.  781.  The  primitives  of  the  language 
include  boolean  and  integer  constants*  free  variables*  equality* 
other  boolean  and  integer  operators*  and  functional  composition. 
Axioms  equate  two  expressions;  the  first  expression  is  generally 
a  function  Composition  and  the  second  expression  is  a  combination 
of  primitives  and  conditional  expressions  like  those  of  ALGOL  60. 
Each  axiom  presented  to  the  DA1STS  processor  is  named  and  has  a 
list  of  the  names  and  types  of  the  free  variables  used  in  the 
axiom.  The  axioms  for  the  bounded  list  operation  DeleteFirst 
might  look  like: 


A * joms 

DeleteFirstl : 

De le te F i r s t (hewLi st )  *  NewList; 

DeleteFirst2(List  AxListl.INT  Axlntl): 

DeleteFirst(AddFirstlAxList1, Axlntl))  * 
if  ListLength(AxListl)  =  11 

then  DeleteLast (AxListl ) 
illfi  *»List1; 

The  axiom  DeleteFirstl  specifies  that  deleting  the  first 
element  in  an  empty  list  results  in  an  empty  list.  DeleteFirst2 
specifies  that  the  result  of  deleting  the  first  element  from  a 

list  to  which  an  element  had  been  added  at  the  head  is  the  same 

as  either:  1)  deleting  the  last  element  from  the  original  list  if 

the  original  list  was  full  (because  adding  an  element  to  the 

front  of  a  full  list  caused  the  last  element  to  be  discarded)*  or 
2)  the  original  list  if  the  original  list  was  not  full  (i.e.»  the 
element  just  added  was  the  one  deleted). 


2.3.  Test  Data 

The  testgoints  section  of  a  DAISTS  program  looks  like  a 
procedure*  complete  with  declarations  and  executable  statements. 
This  section  allows  users  to  build  objects  to  be  referenced  in 
the  subsequent  iSitSStS  section  of  the  program.  An  object  that 
is  expensive  to  construct  can  thus  be  used  in  testing  several 
axioms  without  repeating  its  const ruct i on • 

The  £££££££5  section  of  the  program  contains  a  list  of  axiom 
names  with  values  to  be  substituted  for  the  free  variables  of  the 
axioms.  Sample  t e s£g oi Qli  and  l££l££l£  sections  for  bounded 
lists  of  integers  are  shown  belo*. 


* 
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leSlCSlQ's 

ioi  i 

Cist  Pl,P2 

p1  :=  Aad F i r st ( Add Fi rst ( NewL i st  , 3 ) ,4 ) 

P?  :=  NewList  /*  Initialize  P2  to  be  empty  */ 

I  :=  0 

I  <  11 

~2  P2  :=  Addtast  (P2,I)  /*  Fill  P2  up!  */ 

I  :*  I  ♦  1 

sod 

t esj£et g 

De let e F i rst 2  :  (NewList, 3),  (Pl,5),  <P2»7>, 

(Conc(P1 ,P2) ,44); 

In  the  l£§IB2iQiS  section,  the  two  lists  PI  and  P 2  are 
constructed;  PI  is  (3,  4)  and  P2  is  (0,  1,  ...»  10).  In  the 
testset§  section,  test  data  pairs  are  provided  for  the  axiom 
DeleteF i rst2.  Since  the  axiom  DeleteFirstl  has  no  free 
variables,  no  input  data  can  be  supplied  for  it,  and  DAISTS  will 
generate  one  test  to  see  if  the  implementation  holds  for  it.  The 
four  pairs  of  test  data  for  DeleteFirst?  cause  DAISTS  to  generate 
tests  for  this  axiom  first  with  AxListl  instantiated  as  NewList 
ana  Axlntl  as  3,  then  with  AxListl  as  Pi  and  Axlntl  as  5,  etc. 
For  each  test  set,  the  implementation  is  used  to  evaluate  the 
left  hand  side  of  the  axiom,  and  then  again  to  evaluate  the  right 
hand  side  of  the  axiom,  and  then  the  two  values  are  compared 
using  the  standard  eauality  operator  for  basic  types  of  the 
language  and  a  user-supplier  equality  operation  for  the  abstract 
types.  An  error  message  is  generated  if  the  two  sides  do  not 
agree.  A  restriction  placed  on  the  implementations  for  DAISTS 
(but  not  inherent  in  SIPPL-D)  is  that  the  abstract  operations 
have  to  be  functions  without  side  effects  so  that  the  two 
evaluations  in  an  axiom  do  not  modify  the  test  set  elements. 

2.4.  Run-time  Monitors 

DAISTS  also  generates  code  to  monitor  statement  and 
expression  execution  at  run  time.  All  statements  in  the 
i mp l emen t at i on  and  all  parts  of  the  axioms  must  be  executed  to 


have  a  successful  test.  Furthermore*  all  expressions  in  the 
statements  and  axioms  must  take  on  more  than  one  value; 
user-supplied  equality  operations  are  used  by  the  system  to 
determine  if  objects  with  user-defined  types  change  values, 
unexecuted  statements  or  axiom  fragments*  and  constant  expression 
values  indicate  that  either  simpler  programs  or  statements  can  be 
written  or  more  test  data  needs  to  be  added  to  justify  the 
program's  complexity. 


3.  Methodology 


3 .1 •  Overview 

ke  hypothesized  that  testing  with  formal  specifications 
would  reduce  the  number  of  delivered  errors  without  increasing 
the  cost  of  program  development*  To  test  our  hypotheses*  we 
conducted  an  experiment  where  an  intermediate  class  in 
programming  languages  was  divided  into  two  groups*  given 
identical  English  language  descriptions  of  the  abstract  type 
^list-of-integers'f  and  assigned  to  oroduce  imp lemen t a t i ons  of 
the  type  in  (the  same)  high  level  language*  One  of  the  groups 
used  algebraic  specifications  to  test  and  debug  their 
implementations*  and  the  other  group  used  the  more  traditional 
debugging  method  of  test  programs*  The  axiom  group  was  supplied 
with  the  axioms  for  the  type;  the  control  group  was  supplied  with 
a  test  driver  that  used  the  abstract  operations  to  sort  groups  of 
integers  and  were  allowed  to  produce  other  test  drivers  at  their 
discretion*  Both  groups  had  to  develop  their  own  test  data*  At 
the  end  of  the  experiment*  we  examined  the  projects  that  were 
turned  in  to  discover  residual  errors* 

There  are  several  design  decisions  that  leo  us  to  this 
particular  experiment*  In  evaluating  DAISTS  there  are  realty  two 
issues  to  be  resolved:  the  ease  with  which  users  could  write 
axioms  and  the  ability  of  users  to  develop  programs  from  the 
axioms*  Given  the  relative  inexperience  of  our  subjects 
(primarily  soohmores  and  juniors)*  we  concentrated  on  program 
development  rather  than  specification  development  hoping  to 
justify  a  later*  more  complex  experiment  with  more  sophisticated 
subjects*  Obviously*  if  the  program  development  task  proved  too 
difficult*  the  system^s  worth  would  be  questioned  because  writing 
formal  specifications  before  development  would  only  make  the 
process  more  difficult*  Another  problem  concerned  the  materials 
to  be  given  to  the  subjects*  Providing  the  axioms  to  one  group 
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and  requiring  the  control  group  to  devise  their  own  testing 
programs  seemed  to  make  the  control  group"s  task  more  time 
consuming*  Thus,  we  decided  to  Provide  the  control  group  with  a 
test  routine  (the  sort  program),  which  tested  nine  of  the  twelve 
list  operations.  Since  some  subjects  would  undoubtedly  test  with 
only  the  sort  routine  while  others  would  write  their  own  test 
drivers  (at  least  for  the  three  untested  functions),  we  would  be 
able  to  get  more  detailed  information  about  the  Dehavior  of  the 
control  group. 


3*2*  Choosing  the  groups 

The  class  (of  79  students)  met  together  for  lecture  twice  a 
week,  and  was  divided  into  four  smaller  oroups  that  met  once  a 
week  with  one  of  two  teaching  assistants*  Two  of  the  small 
groups  (one  from  each  assistant)  were  combined  to  form  the  axiom 
group  (45  students),  and  the  other  two  groups  formed  the  control 
group  (34  students)*  Five  of  the  control  group  students,  and 
four  of  the  axiom  group  students  did  not  turn  in  any  project,  and 
were  dropped  from  the  study*  T*o  more  of  the  control  group 
students,  and  one  of  the  axiom  group  students  were  dropped 
because  their  projects  aid  not  appear  to  De  independently 
developed*  Despite  the  fact  that  the  students  were  warned 
several  times  that  every  compilation  was  being  recorded  and  that 
they  were  required  to  work  independently,  three  pairs  of  projects 
are  so  (remarkably)  similar  that  we  could  not  objectively 
consider  them  to  be  independent  efforts  (either  they  started  from 
decks  that  were  equivalent,  and  were  jointly  develocea,  or  one 
deck  started  "development-'  after  the  other  was  completed,  with 
"development"  consisting  of  a  uniform  substitution  of  names  and 
reoraerino  of  code  segments)*  One  o*  each  pair  of  the 
"non-indeoenden t *  projects  (the  "dependent"  one  if  determinable) 
has  been  omitted*  This  left  4H  students  in  the  axiom  group,  and 
27  students  in  the  control  group* 


IQ 


Analysts  of  the  grades  that  they  received  for  the  semester 
showed  only  one  statistically  significant  difference  between  the 
two  groups  -  the  control  group  had  slightly  higher  examination 
scores*  (See  Table  0  below.) 

Table  0  Group  Differences. 

Axiom  Group 

Letter  grade  Mean  Z75JT 

S  t  dev  . 90 

Level  <  SOX 

Project  grade  Mean  120.8 

St  dev  17.8 
Level  <  44X 

Exam  Grade  Mean  41.00 

St  dev  12.12 
Level  <  7X 

Letter  grades  on  a  scale  1=D,  2  =  C»  etc. 

Project  grades  (five  projects  not  counting  experiment) 
on  a  150  point  scale. 

Composite  exam  grades  on  a  100  point  scale. 

Significance  levels  for  a  two*tailed  test. 

3.3.  The  Project 

This  was  the  first  exposure  to  'encapsulated  types'  for 
nearly  all  of  the  students  in  the  class*  so  we  chose  to  assign  a 
rather  small  project  (approx  i  mat  l  y  150  lines)  to  implement 
'bounded  l i s t-of-in t ege rs •  '  All  of  the  operations  on  lists  were 
described  in  English  in  the  project  handout  (see  Appendix  I), 
which  also  contained  instructions  for  using  the  appropriate 
processor.  The  subjects  were  told  that  they  were  to  implement 
all  of  the  functions  described  in  the  handout  and  present  results 
demonstrating  that  the  sort  routine  or  axioms  executed  with  no 


A  manual  describing  the  implementation  language  and  giving 
other  specific  information  (using  the  axioms  for  the  axiom  group 
members,  writing  driver  programs  for  the  control  group)  was  also 
distributed*  The  axioms  and  the  test  driver  program  that  were 
provided  are  in  Appendices  II  and  III  resoe c t i ve l y  • 

Since  the  test  data  had  to  be  submitted  along  with  the 
program  at  compilation  for  the  axiom  group  (to  allow  DAISTS  to 
generate  its  test  driver)f  we  felt  that  the  control  group  should 
also  be  required  to  submit  their  test  data  with  their  compilation 
requests  to  make  the  development  environments  more  nearly 
eauivalent • 

we  also  provided  separate  lecture  and  lab  meetings  for  the 
two  groups*  and  tried  to  exchange  experimenters  so  that  each 
group  met  with  each  experimenter  to  nullify  any  bias  our  lectures 
could  be  giving*  The  students  were  informed  that  they  had  been 
divided  into  two  grouos*  and  were  asked  not  to  exchange 
information  about  how  the  two  groups  were  different*  However* 
since  the  abstract  type  that  they  were  implementing  was  the  same 
for  Doth  aroups*  some  of  the  details  of  the  implementation 
language  were  discussed  with  both  groups  present* 


3*4*  Data  Collection 

A  special  processor  was  set  up  to  limit  access  to  DAISTS 
it  did  not  allow  members  of  the  axiom  group  to  write  separate 
test  drivers*  and  it  did  not  allow  the  members  of  the  control 
group  to  use  the  axioms*  This  processor  also  saved  a  copy  of  a 
every  deck  submitted*  DAISTS  was  hidden  so  that  the  students 
were  forced  to  go  through  the  orocessor  (so  that  all  submissions 
could  be  recorded)*  and  the  students  were  told  that  their  aecks 
were  being  collected*  This  approach  led  to  a  number  of  identical 
decks  being  saved  by  the  submission  processor  -  a  student  would 
run  a  deck  at  his  terminal  and  then  run  the  aeck  again  to 
generate  a  listing  on  the  printer*  or  programs  failed  to  complete 
execution  before  their  system  default  time  limit  was  exhausted  so 
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the  oecks  were  resubmitted  with  larger  time  limits. 

3.5.  Identifying  errors 

After  all  of  the  students"  projects  were  collected  and 
graded,  the  files  of  deck5  were  examined.  For  each  student,  the 
deck  that  corresponded  to  the  listing  that  was  turned  in  was 
separated  into  the  class  definition  and  the  debugging  data.  We 
then  debugged  each  implementation,  both  by  "desk  checking"  and  by 
using  the  DAISTS  system,  with  a  targe  variety  of  data  points  for 
each  axiom.  Many  of  the  errors  that  we  found  were  detected  by 
turning  on  the  subs c ri p t -c hec k  i  ng  feature  of  the  compiler  - 
apparently  very  few  of  the  students  used  this  feature. 

As  we  were  debugging  their  imp lement at  ions ,  we  found  that 
several  student's  implementations  had  "subjective  restrictions' 
that  were  not  clearly  specified  in  the  project  assignment.  These 
"subjective  restrictions"  could  be  interpreted  as  errors,  but  a 
case  could  be  made  for  allowing  them  as  correct  restrictions. 
One  student  stored  only  single-digit  positive  integers.  Several 
students  could  correctly  store  any  integers  except  zero,  which 
they  used  as  markers  in  their  representations. 

when  these  "subjective  errors"  were  counted,  the  results 
were  not  substantially  different  from  the  results  reported  below, 
which  come  from  only  counting  the  "objective  errors"  -  code  that 
fails  no  matter  how  favorable  the  input  values  selected. 


3.6.  Measuring  errors 

We  also  faced  a  dilemma  in  chosing  which  errors  to  count, 
and  how  to  report  them.  we  feel  that  the  most  conservative 
objective  measure  that  we  can  use  is  fyQ££ions  £201212109  £££&££• 
If  a  function  in  the  submitted  project  had  to  be  changed, 
regardless  of  the  number  of  changes  that  had  to  be  made  to 
correct  the  function,  it  was  counted  as  a  single  "function 
containing  an  error."  We  like  the  resolution  of  this  measure, 
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because  1)  it  is  more  nearly  representation-independent  than  any 
error  measure  that  is  influenced  by  the  structure  of  the  code  of 
the  implementation*  and  2 )  the  project  description  defined 
functions*  the  axioms  specified  functions*  and  the  sort  routine 
used  the  specified  functions. 

we  compared  our  measure  to  that  of  CGannon  773  who  reported 
3i£l2Q£l  £££2£S  and  errgr  2££y£££Q£££  where  (for  example)  if  the 
same  error  in  computing  the  length  of  a  list  was  made  in  three 
places*  it  would  count  as  one  distinct  error*  but  three  error 
occurrences.  Our  data  produced  similar  results  for  both  of  these 
measures  and  for  the  measure  'functions  containing  errors.' 

Several  of  the  students  made  errors  in  the  selection  of 
their  representations.  One  student  used  a  circular  list  and  had 
an  ambiguity  in  his  representation  so  that  a  full  list  was  not 
distinguishable  from  an  empty  one.  This  error  could  only  be 
fixed  by  adding  a  word  to  his  representation  and  repairing  many 
of  the  functions.  Another  student's  project  was  corrected  by 
merely  changing  the  size  of  an  array  in  his  representat ion  (no 
functions  needed  changing).  These  decks  were  charged  with  one 
incorrect  function  to  account  for  the  change  to  the 
represent  at  ion  (in  addition  to  the  incorrect  functions  that  they 
were  charoed). 

In  the  results  reported  below*  the  measure  functions  that 
cfiQtaiQgd  abi££ii££  £££2££  «as  used. 

3.7.  Measuring  Cost  of  Development 

It  is  inherently  difficult  to  measure  orogrammer  effort.  It 
is  especially  difficult  to  measure  effort  o *  students  who  do  not 
work  regular  schedules  and  who  are  not  inclined  to  keep  track  of 
efforts  on  a  project  near  the  end  of  a  semester.  Since  the  only 
enforcable  metric  which  we  could  employ  was  the  number  of  runs 
submitted*  and  since  the  previously  mentioned  duplicated  decks 
involved  none  of  the  debugging  effort  which  we  were  tryinq  to 
measure*  the  most  convenient  and  consistent  measure  that  we  can 


use  is  Qgmfcjj  g  f  31iiiQ£i  rgng 


3*8*  Statistical  techniques 

we  chose  to  use  the  Mann-whitney  U-test  CSiegel  563  for 
doing  the  analysis  of  the  data  froa  our  experiment*  The 
Nann-whitney  U-test  is  non-peramet ric*  and  our  data  is  (at  best) 
an  ordinal  measure  of  performance.  Parametric  tests  also  require 
that  the  samples  be  drawn  *  rom  an  uniform  distribution*  which  we 
cannot  guarantee  for  our  data. 
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<*•  Results 


We  report  the  data  both  for  the  subgroups  that  successfully 
completed  the  project  and  tor  the  entire  groups*  By  successfully 
completing  the  project*  we  mean  only  that  the  output  of  the 
program  that  was  submitted  for  grading  displayed  no  obvious 
errors*  many  of  the  students  turned  in  projects  that  they  knew 
were  not  correct  -  students  in  the  aaiom  group  had  error  messages 
complaining  about  inconsistent  axioms*  and  students  in  the 
control  group  turned  in  projects  for  which  the  sort  routine  would 
not  correctly  sort  the  integers  that  they  used  to  demonstrate 
that  their  orograms  worked.  In  the  axiom  group*  32  out  of  40  who 
turned  in  the  project  successfully  completed  it*  and  in  the 
control  group  22  out  cf  27  were  successful* 

we  have  subdivided  the  successful  control  group  into  two 
groups  for  further  comparison:  one  that  wrote  and  ran  small  test 
driver  programs  in  addition  to  running  the  sort  program  (which  is 
more  like  an  integration  test  than  a  driver)*  and  another  group 
that  used  the  sort  routine  exclusively  for  testing*  Of  the  22 
successful  control  group  members*  7  wrote  their  own  driver 
program? 

Since  the  sort  program  tested  9  of  the  12  list  operations* 
we  have  reported  both  the  number  of  incorrect  functions  tested  by 
the  sort  routine  and  the  total  number  of  incorrect  functions* 
The  subjects  were  required  to  implement  all  the  functions  and 
were  encouraged  to  write  extra  functions  to  display  list  objects 
as  a  debugging  aid* 

In  the  tables  below*  we  report  the  means  and  standard 
deviations  (following  the  means  in  parentheses)  of  the  number  of 
incorrect  functions  tested  by  both  the  axioms  and  the  sort 
routine*  the  total  number  of  incorrect  functions*  and  the  number 
of  dist inct  runs* 
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4.1.  All  Subjects 


All  subjects  in  both  groups  had  similar  numoers  of  incorrect 
functions  tested  by  the  sort  routine  and  distinct  runs.  while 

the  means  favored  the  members  of  the  axiom  group  Can  average  of 

.18  fewer  incorrect  functions  out  of  the  9  functions  tested  by 

the  sort  routine  and  .71  fewer  distinct  runs),  the  differences 

were  not  significant.  As  expected.  the  axiom  group  did 
significantly  better  than  the  control  group  in  eliminating  errors 
in  all  the  functions. 

Table  I  All  Axiom  (40)  and  Control  (27)  Subjects 

Axiom  group  Sort  group  Level 

Incorrect  sort  functions  .60(1.18)  .78(1.64) 

All  incorrect  functions  .82(1.51)  1.78(2.20)  <.QQ3X 

Distinct  runs  11.77(5.65)  12.48(8.53) 


4.2.  Successful  Subjects 


when  we  consioer  only  those  students  who  successfully 
completed  the  project,  the  axiom  group  did  marginally  better  than 
the  control  group  even  on  the  functions  tested  by  the  sort 
routine.  This  result  appears  despite  the  fact  that  the  sort 
routine  did  do  a  fairly  good  job  of  exposing  the  errors  in  these 
routines  for  those  subjects  choosing  good  sets  of  data  to  use 
with  it  (in  the  sense  that  when  data  for  the  sort  prooram 
contained  all  of  the  boundary  cases  of  the  sort  routine.  all  of 
the  boundary  cases  of  the  list  functions  were  tested).  Of 
course,  the  results  are  even  more  striking  when  we  consider  all 
functions  that  were  assigned.  The  axiom  group  delivered  more 
correct  functions  than  did  the  control  group  while  taking  fewer 


r  uns . 


Table  II  Successful  Axiom  (32)  and  Control  (22)  Subjects 


Axiom  grouo  Sort  group  Level 


Incorrect  sort  functions  .12(  .33)  .23(  .42) 
All  incorrect  functions  .191  .39)  1.23(1.20) 
Distinct  runs  10.97(5.60)  11.41(7.60) 


<20X 
<  •  0Q4X 


4.3'.  Subjects  Writing  Their  Own  Drivers 

we  expected  that  those  subjects  in  the  control  group  who 
wrote  driver  programs  in  addition  to  using  the  sort  routine  would 
test  as  effectively  as  the  axiom  group,  but  would  require  more 
runs  to  debug  their  own  drivers.  The  data  in  Taole  III  suoports 
these  hypotheses,  except  when  we  consider  all  the  functions 
assigned.  Even  those  subjects  writing  their  own  driver  programs 
did  not  produce  as  many  correct  functions  as  the  axiom  grouo  did. 

Table  III  Successful  Axiom  (32)  and 

Driver-Writing  (?)  Subjects 

Axiom  group  Driver  group  Level 

Incorrect  sort  functions  .12(  .33)  .14(  .35) 

»ll  incorrect  functions  .19(  .39)  . 5 7 (  .49)  <2X 

Distinct  runs  10.97(5.60)  15.14(5.82)  <4X 

Examining  the  runs  of  the  dr i ver-wr i ti ng  subjects  to 
determine  why  their  efforts  did  not  match  those  of  the  axiom 
group,  we  find  a  distinct  lack  of  testing  discipline.  Five  of 
the  7  subjects  had  test  drivers  that  could  exercise  all  the 
functions  (one  subject  missed  one  function  and  the  other  subject 
missed  two).  Four  of  the  subjects  used  effective  tests,  trying  a 
variety  of  objects  in  different  operations,  while  while  two  other 
subjects  with  extensive  test  drivers  just  did  not  seem  to  use 
enough  data  to  cover  the  necessary  cases.  Four  of  the  subjects 
used  drivers  before  using  the  sort  routine  seriously  as  an 
integration  test.  (l.e*.  they  may  have  used  it  to  compile  their 
implementations  initially,  but  did  not  try  to  debug  using  it.) 
Two  other  subjects  used  drivers  only  in  response  to  specific 
errors  that  occurred  in  debugging  with  the  sort  routine. 


4.4.  Subjects  testing  with  the  Sort  Program  Only 


Those  subjects  testing  only  with  the  sort  progra*  used  an 
average  of  1.3  fewer  runs  than  did  the  members  of  the  axiom 
group*  but  the  axiom  group*  but  did  not  produce  as  many  working 
functions  even  when  we  consider  only  the  functions  tested  by  the 
sort  program  (8.73  to  8.88).  Part  of  the  explanation  tor  this 
result  may  be  that  the  sort  program  did  not  encourage  the  members 
of  the  control  qrouo  to  test  more  thoroughly*  The  sort  program's 
effectiveness  as  a  testing  vehicle  was  impaired  by  poor 
selections  of  test  data  that  did  not  include  the  boundary  cases 
of  the  sort's  domain  (e*g*«  empty  lists*  lists  with  duplicate 
members*  etc.). 

Table  IV  Successful  Axiom  132)  and 

Sort-Only  (15)  Subjects 


Axiom  group 

Incorrect  sort  functions  .1?(  *33) 
All  incorrect  functions  .19(  »39) 
Distinct  runs  10*97(5*60) 


Sort  group  Level 

,27 (  .44)  <14X 

1.53(1.31)  <.002X 

9.67(7.70)  <8X 


5.  Conclusions 

We  have  shown  that  DAISTS  can  encourage  even  inexper ienced 
users  to  develop  effective  tests  for  their  implementations. 
Those  subjects  who  used  only  the  sort  program  to  test  their 
implementations  stopoed  testing  too  soon  because  the  data  they 
ted  the  sort  program  did  not  expose  errors  in  their  list 
i mp l emen t at i ons •  The  axiom  group  needed  more  runs  to  satisfy 
0  A I S  T  S  t  out  correctly  developed  more  of  the  functions  used  by  the 
sort  program. 

The  discipline  of  testing  with  DAISTS  can  help  users  avoid 
less  systematic  testing  methods.  Even  if  we  consider  only  the 
subjects  in  our  study  who  wrote  their  own  test  drivers*  we 
observe  a  variety  of  Questionable  testing  practices  -  omitted 
functions*  failures  to  consider  boundary  cases*  and  generally 
insufficient  test  data.  The  formal  specification  required  by 
DAISTS  identifies  the  boundary  cases  and  clearly  defines  their 
treatment.  Furthermore*  DAISTS  run-time  monitoring  routines 
ensure  that  the  code  handling  boundary  cases  is  exercised. 

Performing  this  type  of  study  can  also  give  us  insights  that 
help  us  improve  our  system.  We  were  frustrated  oy  the  ambiguity 
that  the  students  read  into  our  careful  English  descriptions  (the 
^subjective  errors4'  that  we  identified  -  single  digit  integers* 
using  aero  for  a  marker*  etc.)*  but  such  imprecision  is  inherent 
in  informal  specifications.  Even  including  formal  specifications 
for  the  subtypes  used  in  building  the  new  type  does  not  prevent 
the  omission  of  test  data  that  exposes  the  confusion.  We  feel 
that  DAISTS-like  systems  might  expose  these  errors  with 
special-values  testing  s t rat  eg  ies.  CHowden  783*  e.g.*  adding  test 
sets  that  include  the  constant  functions  of  the  subtypes  (0  for 
22is*  null  and  blank  strings*  NewList*  EmptyStack*  etc.)  and  the 
constants  of  the  subtypes  that  appear  in  the  text  of  the 
iffplementat  ion. 
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This  experiment  did  not  evaluate  the  subject's  ability  to 
unite  specifications*  While  many  programmers  might  have 
difficulty  producing  axioms  without  training*  we  feel  that  this 
fact  does  not  render  the  tool  useless*  Our  own  experience  in 
teaching  programmers  to  write  algebraic  axioms  leads  us  to 
conclude  that  they  are  not  as  cumbersome  as  many  believe* 
Another  experiment  is  needed  to  test  the  validity  of  this 
hypothesis  • 

The  computer  science  community  has  reached  a  consensus  on 
the  desirability  of  requirements  analysis  and  formal 
specifications*  Having  a  tool  which  can  incorporate 
specifications  into  the  development  process  will  provide  the 
motivation  and  experience  necessary  to  use  them.  writing  formal 
specifications  need  not  oe  considered  overhead  if  they  can  reduce 
the  effort  needed  to  write  and  debug  test  driver  programs* 
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Append i  *  I 


first  page  of  handout 


You  are  to  write  the  CLASS  implementation  for  lists  of 
integers*  A  list  is  an  ordered  collection  of  elements  which  may 
have  elements  added  and  deleted  at  its  endsf  but  not  in  its 
middle*  The  operations  that  you  must  "export"  are:  AddFirst, 
AddLast*  Cone,  OeleteFirst,  OeleteLast,  First,  IsEmpty, 
Listtqual,  Listbength,  NewList,  and  Reverse*  Each  operation  is 
described  in  detail  below* 

The  lists  are  to  contain  up  to  eleven  (11)  elements*  If  .an 
element  is  added  to  the  front  of  a  ••full"  list  (one  containing 
eleven  elements  already),  the  element  at  the  back  of  the  list  is 
to  be  discarded.  Elements  to  be  added  to  the  back  of  a  full  list 
are  discarded*  Requests  to  oelete  elements  from  empty  lists 
result  in  empty  lists,  and  requests  for  the  first  element  of  an 
empty  list  results  in  zero  (Q)* 

Remember  that  the  operations  that  you  implement  are  to  be 
functions,  and  that  they  may  ***N0T***  change  their  parameters! 
If  a  function  needs  to  manipulate  a  parameter  to  perform  the 
operation,  the  parameter  is  to  be  COPIED  to  a  LOCAL  variable 
BEFORE  the  change  is  performed!  You  may  use  any  reoresentat ion 
you  choose  to  implement  your  lists*  The  detailed  operation 
descriptions  are  below: 


List  FUNC  Addf i rs t ( L i st  L,INT  I)  -  Returns  the  list  with  I  as 
its  first  element  followed  by  all  of  the  elements  of  L*  If 
L  is  “full"  to  start,  L^s  last  element  is  ignored* 

List  FUNC  Addlast(List  L,INT  I)  -  Returns  the  list  with  all  of 
the  elements  of  L  followed  by  I.  If  L  is  full  to  start,  I 
i$  ignored* 

List  FUNC  Conc(List  Ll>List  L 2)  -  Returns  the  list  made  up  of 
the  elements  of  list  LI  followed  by  the  elements  of  l2*  If 
Li  and  L2  together  contain  more  than  eleven  (11)  elements* 
then  the  extras  are  to  be  ignored* 

List  FUNC  De letef i rs t  (L i st  L)  -  Returns  the  list  containing 
all  but  the  first  element  of  L.  If  L  is  empty,  then  it 
returns  an  empty  list* 

List  FUNC  Deletelast (List  L)  -  Returns  the  list  containing  all 
but  the  last  element  of  L*  If  L  is  empty,  then  it  returns 
an  empty  list* 

INT  FUNC  FirstlList  L)  •  Returns  the  first  element  in  L*  1 4  L 
is  empty,  then  it  returns  zero  (0) * 

INT  FUNC  Isempty(List  L)  ~  Returns  one  (1)  if  L  is  empty*  zero 
(0)  otherwise* 

INT  FUNC  L i s tequa l <L i st  Li, List  LZ)  -  Returns  one  (1)  if  the 
two  lists  are  element  for  element  equivalent  (e.g* 
First(Ll)  -  Fi rst < LZ )*•••) *  and  zero  (0)  otherwise*  Note 

that  two  empty  lists  are  considered  equal* 

INT  FUNC  L i s 1 1 eng t h  (l  ist  L)  -  Returns  the  count  of  elements  in 
L*  An  empty  list  has  a  count  of  zero  (Q)  elements* 

List  FUNC  Newlist  -  Returns  a  list  initialized  to  be  empty 

List  FUNC  ReverseCList  Li)  -  Returns  a  list  containing  the 
elements  of  Li  in  reverse  order. 
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Append**  I 


second  page  o*  assignment  for  control  group 


A  iest  routine  has  been  written  for  you,  or  you>  may  write 
your  own  test  routines*  The  provided  routine  reads  in  groups  of 
integers,  sorts  them,  ana  prints  out  the  smallest,  11  of  each 
group*  The  test  routine  exoects  the  groups  of  integers  to  be 
separatee  by  zero,  A  sample  test  run  usmo  the  provided  test 
routine  is  shown  below: 


3a  dd  simold*project*  setup 


<done  once  per  run> 


3$  •  S IMPLD , S  <calls  the  compiler, 

asks  for  listing> 

<list  i mp lemen t a t ion>  <your  CLASS  for  list$> 

f T E S T  <causes  the  test  routine 

to  be  provided> 

<groups  of  integers,  separated  oy  zeros> 

5>eof  <end  of  the  data> 

The  data  for  the  test  routine  may  have  any  number  of  integers  or 
groups  of  integers  per  card,  with  the  integer  0  separating  each 
group*  Soaces  are  used  to  separate  the  integers  when  more  than 
one  integer  is  on  a  card# 

A  sample  run  for  using  your  own  test  routine  is  shown  below: 

•  S IMPLD  ,  $  <call  compiler  as  above, 

assuming  setup  is  done> 

<  l  i  s  t  imnlementation> 

<your  test  driver> 

5  D  A  T  A 

<your  data> 

Seof 


You  will  be  required  to  submit  your  list  implementation  via 
the  deck  submission  processor # (to  be  discussed  in  class),  and  you 
will  also  turn  in  a  listing  of  a  run  using  the  provided  test 
routine,  that  shows  several  groups  of  integers  correctly  sorted* 


c  c 


Appendix  I 


second  page  of  assignment  for  axiom  group 


Axioms  have  been  written  which  you  must  use  to  debug  your 
CLaSS.  You  may  add  axioms  of  your  own  at  your  discretion.  A 
sample  run  is  shown  below: 

2add  s i mp ld*pr o j ec t • se t uo  <done  once  per  run> 


S$.$IMPLD*$  <calls  compilerr  asks 

for  listina> 

<list  imp lemen t at ion>  <your  CLASS  for  lists> 

SAXIOMS  <causes  axioms  to  be  provided> 

<your  optional  axioms> 

TeSTPOINTS 
<your  testpoints> 

testsets 

<your  testsets> 

START 

3eof  <that's  all  you  need> 


You  will  be  required  to  submit  your  list  implementation  via 
the  deck  submission  processor  (to  be  discussed  in  class)*  and  you 
ill  also  turn  in  a  listing  of  a  run  using  the  provided  axioms* 
ith  no  axiom  failures  and  all  statements  executed. 
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Appendix  II  The  Axioms  supplied  to  the  axiom  group 


A SlSSi 

/*  These  axioms  are  constructeo  following  the  6uttag 

rules  for  deciding  which  axioms  need  to  be  constructed* 
The  functions  in  the  **0"  group  are: 

IsEmpty,  ListEoual,  ListLength,  First 
The  functions  in  the  "toil”  aroup  are: 

NewList,  AddFirst 

The  functions  in  the  "T0I2"  group  are: 

AddLast*  DeleteLast,  Deletepirst,  Cone,  Reverse 


I sEmptyl : 

I sEmpt y (NewL  i  s t >  =  1; 


I sEmpty2 (L  i  st  AxList1»int  Axlntl): 
IsEmpty(AddFirst(AxList1  , Axlntl  )) 


0; 


List  Equa  11  : 

List  Equal (NewList , NewList)  =  1; 

L is t Equa 12 (L i s t  AxList1,int  Axlntl): 

L ist Equa l (NewL i st , AddFTr st ( A xLi st 1 , Ax Int 1 ) )  =  0; 

L i s t Equa 13 (L i St  AxList1,int  Axlntl): 

L  i  st  Equa  l  ( AddF  i  rst  (Ax  lTs  1 1 ,  Ax  mt  1 ) ,  NewL  i  s t )  =  0; 

L ist Equa 14 (List  AxList1,List  Axlist2,int  Axlntl, int  Axint2): 

L  ist  Equa l ( AddF i rst ( Ax  Li st 1 , Ax Int 1 )7*3dF i rs t (A  xCi s 1 2  » Ax int 2 ) ) 
if  Axlntl  <>  Axlnt2 
Jhen  0 
else 

"TT”L i stiengt h( AxL i s 1 1  )  *  11 

then  7*  Need  to  trim  the  end  off!  */ 

'CTstEqua l(DeleteLast(AxList1),0eleteLast(AxList2)> 
else  /*  Compare  them  just  as  they  are!  */ 

“Cist  Equa l(AxList1,A»List2); 


L istLengthl : 

ListLergth(NewList  )  =  Q; 

ListLength2(List  AxList1,int  Axlntl): 

ListLength(AddFirst(AxCist1  , Axlntl))  = 
if  L l stLengt h (A xLi st 1  )  =  11 
then  11 

gill  1  ♦  Li st Lengt h ( Ax  Li st 1 ) ; 


F irst  1 : 

First (NewLi st )  =  0  ; 


First2(List  AxList1,int  Axlntl): 
First(AddFirst(AxCTst1, Axlntl)) 


Axlntl  ; 


/*  Now  for  the  MT0I2M  function  definitions:  */ 
AddLast1(int  Axlntl): 

AddLast ThewL i st , Ax Int 1 )  =  AddF i rst (NewL i s t ,  Ax Int 1 ) ; 

AddLast 2  (L  i  st  AxList1,int  Axlntl, int  Axlnt2): 
AddLast(AddFirst(AxCTst1,AxIntT7,AxInt2)  = 
AddFirst(AddLast(AxList1 ,AxInt2) , Axlntl); 


DeleteLastl : 

DeleteLast (NewL  ist  )  *  NewList; 

De l e t eLast 2 (Li st  AxList1,int  Axlntl): 

DeleteLast(AddFirst(AxCTst1,AxintD)  = 
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Appendix  II  The  Axioms  supplied  to  the  axiom  group 


if  I$Empty(AxLi stl ) 

””ih£Q  NewList 

^Li stLeng t hC A xli s tl )  s  11 

t hgn  AddFir$t(peleteLast(DeleteLast(AxList1))»MxInt1) 
£iS£  A ddFirst(DeleteLast( AxListl), Axlntl); 


Delet  eF i r st 1 : 

De let eF i rst ( NewLi s t )  =  NewList; 

Del eteF i r st 2 (L i st  AxListl, in£  Axlntl): 

DeleteFirstCAddFirstfAxCTstl.Axintl))  = 
if  ListLength ( A xLi stl  )  *  11 
thgn  De le teLa st ( axl i st  1 ) 

£i££  A xti stl; 


ConcKList  AxListl): 

Cone (NewL i st , A xLi s tl )  *  AxListl; 

Conc2(List  AxListl, List  AxList2.int  Axlntl): 
Conc(AddFirst(AxList1yAxInt1)7KxList2)  = 
AdoFirst(Conc(AxList1,AxList2), Axlntl); 


Reversel : 

Reverse (NewList  )  »  NewList; 

R eve r se2 (L i s t  AxList1»int  Axlntl): 

Reverse(AddFirst(AxCist1. Axlntl))  = 
if  ListLength(AxList1 )  =  11 

then  AddLast(Peverse(DeleteLast(AxList1))yAxInt1) 
else  AddLastlReverse(AxListl), Axlntl); 


Appendix  III  The  sort  routine  given  to  the  control  group 


B£2£  Main  /*  The  driver  for  the  sort  program  to  test  lists.  */ 

/*  Read  in  a  series  of  numbers  that  ends  with  teroi  sort  them,  */ 
/*  and  then  print  out  the  smallest  ListSize  of  them,  */ 

int  Holder  /*  A  place  to  read  numbers  into  •/ 

151  Setnumber  /*  The  numoer  of  the  Current  set  */ 

inf  Counter  /*  A  counter  of  the  number  of  numbers  in  Unsorted  */ 

Cist  Unsorted  /*  where  the  unsorted  numbers  are  read  into  */ 

List  Sorted  / *  Where  the  sorted  numbers  are  stored!  */ 

/*  loop  for  reading  in  sets  of  numbers  */ 

Setnumber  :=  1  /*  Start  to  work  on  the  first  set  */ 
while  .ngj.  eoi  dg 

Sorted  I-"Newiist  /*  No  sorted  numbers  in  tnis  group  */ 
Unsorted  :=  NewList  /*  Also  no  unsorted  numbers  in  yet!  •  / 
regd(Holder)  /*  To  initialize  Holder!  */ 

unite  Holder  <>  C  .ana. 

.n o£.  egi  go  ““7*  Keep  reading  and  sorting  */ 

Counter  :*  (T  /*  Set  to  count  the  unsorted  list  */ 
while  Holder  <>  0  .and. 

"Counter  <  ListSize  .and.  .not.  eoi  do 
Unsorted  :=  AadFi rstTUnsorFe3,HotderT 
Counter  :*  Counter  ♦  T 

read(Holder)  /*  Get  the  next  number  of  the  set  */ 

end 

/*  Either  unsorted  is  full,  or  this  set  is  finished!  */ 
/*  Must  first  join  unsorted  numbers  with  sorted  ones  */ 

Sorted  :=  Merge (Sort ed , Sort (Unsort ed) ) 

end 

/*  Here  we  must  have  hit  an  end  of  a  set!  */ 

/*  Print  out  the  first  "ListSize"  worth  of  numbers  */ 

wr i t e (sk io ♦ 'Sort ed  numbers  of  set  number"  .Setnumber ) 
wliTte  .not.  I  sEmcty  (So  r  ted )  do 

"wr iTe ( F i rst (Sort ed) )  /^Output  smallest  number  in  set*/ 
Sorted  : =  Deletefi rst ( Sort ed) 

end 

SeFnumber  :  *  Setnumber  ♦  1  /*  Now  start  the  next  set  */ 

jng 

wrile(£k ig.ikig.skig, "Out  of  sets  of  numbers  to  sort") 


List  fync  Merge(List  Mlin.List  *2in)  /‘merges  two  sorted  lists*/ 
List  Result 

List  Ml , M2  /*  Locals  so  that  we  do  not  change  the  parameters!  */ 


too!  */ 


r=  Mlin  /*  Copy  parameter  */ 

:=  M2in  /*  Copy  second  parameter 
ult  : =  NewList 

i£  •  Qfi£»  iIsEmpty(¥1)  »gr.  IsEmpty(M2>) 
o  /*  Done  when  one  is  empty!  */ 

"  if  F i rst (Ml )  <*  Fi rst  0*2) 

""then  /*  Take  next  value  from  Ml  */ 

““Result  : =  A ddF i r s t (Resul t ,  F i rst (Ml  )  ) 

Ml  :*  Oe let eF i rst (Ml )  /*  Don"t  need  fi 
5lje  /*  Take  from  *>2!  */ 

Result  :*  A ddF i rs t ( Resul t , F i rs t (m2)  ) 

M2  :=  Oe let eF i rs t  (m2  )  /*  Discard  first  after  copying  */ 

J  £02 


rst  number  *  / 


/*0ne  of  the  two  lists  is  empty  -  catenate  them  all  together!*/ 
££iU£Q( Cone ( Reverse ( R esu 1 1 ), Cone (M i  ,  M2) ) )  /*and  reorder  Result'*/ 
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Appendix  III  The  sort  routine  given  to  the  control  group 


£££  List  Tyne  SortCList  Inlist)  /*Sorts  into  increasing  order*/ 

/*  This  procedure  vorks  by  a  merge  sort  -  Split  the  list  in  */ 
/*  two,  sort  each  halt,  and  then  merge  the  two  sorted  halts!  */ 

List  Haiti, Half2 


Haiti  :*  NewList  /*  Initialize!  */ 

Halt2  :*  Inlist  /*  Initialize!  */ 

ytiifi  ListLength(Haltl)  <  ListLeng th(Ha l f 2 >  do 


end 

ISlU£Q<!**r9*  (Sort  (Haiti)  .Sort  <Half  2))> 
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